Messaging System For A Virtual Assistant

ABSTRACT

Messaging with a virtual assistant by: receiving a first request (e.g., spoken by a user to a virtual assistant device hosting the virtual assistant) to check for messages received by an electronic device and transmitting the first request to a backend server for processing. The backend server requests data associated with the messages from the electronic device and, upon receiving the data, generates a response to the first request based on the received data. The backend server then transmits the response to the virtual assistant and the virtual assistant communicates the response (e.g., using audio produced by the virtual assistant device). The virtual assistant receives a reply from the user to the message. The reply is transmitted to the backend server for processing. The backend server transmits the reply to the electronic device which automatically transmits the reply to a recipient of the reply (e.g., a sender of the message).

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. provisionalapplication Ser. No. 62/939,516, filed on Nov. 22, 2019, U.S.provisional application Ser. No. 62/961,958, filed on Jan. 16, 2020,U.S. provisional application Ser. No. 63/014,577 filed on Apr. 23, 2020,and U.S. provisional application Ser. No. 63/019,124, filed on May 1,2020, the contents of each of which are hereby incorporated byreference.

TECHNICAL FIELD

The present disclosure relates generally to user access to messagesreceived by an electronic device, and more specifically to utilizing avirtual assistant to assist the user with sending/receiving messagesthrough the electronic device.

BACKGROUND

Many electronic devices (e.g., a smart phone) provide users with theability to communicate with each other by transmitting and receivingmessages. For example, the electronic devices may include texting,e-mail and/or instant messaging capabilities that enable the users totransmit and receive text, images, sounds, etc. The user may interactwith these electronic devices in many different ways, such as throughmechanical devices (e.g., keyboards, mice, etc.), touch screens, motionand gesture capture, etc. However, another way to interact withelectronic devices is through a virtual assistant, e.g., via spokenverbal input.

Using spoken verbal input to interact with electronic devices becomesmore complicated when the user attempts to discover, pair, and/orconfigure a secondary electronic device to be controlled by a primaryelectronic device that provides functionality for receiving spokenverbal input to control both the primary electronic device and thesecondary computing device.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. Some embodiments are illustrated by way of example, and notlimitation, in the figures of the accompanying drawings.

FIG. 1 is a use-case diagram showing a system, consistent with someembodiments, for messaging over a network by using a virtual assistant.

FIG. 2 is a block diagram illustrating an example of a messagingapplication, consistent with some embodiments, for messaging over acommunication network by using a virtual assistant.

FIG. 3 is a block diagram illustrating an example of a virtual assistantapplication, consistent with some embodiments, for messaging over acommunication network by using a virtual assistant.

FIG. 4 is a ladder diagram illustrating a method, in accordance withsome embodiments, for accessing messages over a network by using avirtual assistant.

FIG. 5 is a ladder diagram illustrating a method, in accordance withsome embodiments, for sending messages over a network by using a virtualassistant.

FIG. 6 is a flow diagram illustrating a method, in accordance with someembodiments, for accessing messages over a network by using a virtualassistant.

FIGS. 7A and 7B are flow diagrams illustrating example methods for voiceprocessing to access messages over a network by using a virtualassistant.

FIG. 8 is a flow diagram illustrating a method, in accordance with someembodiments, for sending messages over a network by using a virtualassistant.

FIG. 9 is a flow diagram illustrating a method, consistent with someembodiments, for requesting message notifications by using a virtualassistant.

FIG. 10 is a diagrammatic representation of machine, in the example formof a computer system, within which a set of instructions may be executedto cause the machine to perform any of the methodologies discussedherein.

DETAILED DESCRIPTION

A virtual assistant may take the form of an application, module,software, etc., capable of analyzing input (e.g., verbal input) toidentify a user's intent and then provide assistance to the user basedon the identified user intent. For example, a virtual assistant may bean application such as ALEXA (made available by Amazon.com Inc., ofSeattle WA) that converts verbal input to text (e.g., using a voiceprofile for a known user) and then uses semantic analysis to analyze thetext, recognize meaningful terms in the text and then determine the userintent based on data associated with the meaningful terms. A virtualassistant, like ALEXA, may be voice-controlled to assist the user withplaying audio, answering questions, engaging the user's favoriteservices (e.g., a weather application), or even controlling homeappliances or other electronic devices (e.g., a smart phone).

SUMMARY

Provided is a computer implemented method to be carried out with aprocessor and a memory, comprising: a) receiving a message on a firstdevice; b) receiving an instruction to deliver the message via a virtualassistant; c) communicating the message from the first device to aserver of the virtual assistant; and d) the virtual assistant thendelivering the message to the user. The virtual assistant cancommunicate the message via a virtual assistant device.

Provided is a computer implemented method to be carried out with aprocessor and a memory, comprising: a) receiving a message on the firstdevice; b) receiving an instruction to notify the user of the messagevia a messaging application on the first device; c) receiving a tokenfrom the server associated with the virtual assistant; d) matching theuser of the virtual assistant and the messaging application based on thetoken; e) communicating the notification to the server of the virtualassistant; and f) the virtual assistant then delivering thenotification(s) to the user via the virtual assistant device (e.g., viaa blinking light).

Provided is a computer implemented method to be carried out with aprocessor and a memory, comprising: a) receiving a message on the firstdevice; b) receiving an instruction to notify the user of the message;c) storing the message (such as for less than 24 hours) to a serverassociated with the messaging application; d) the virtual assistant thendelivering the notification(s) to the user via the virtual assistantdevice; e) receiving an instruction to deliver the message via thevirtual assistant; f) communicating the message to the server of thevirtual assistant; and g) the virtual assistant then delivering themessage to the user.

Provided is a computer implemented method to be carried out with aprocessor and a memory, comprising: a) receiving a message on a firstdevice; b) receiving an instruction to deliver the message via thevirtual assistant; c) communicating the message to the server of thevirtual assistant; d) the virtual assistant then delivering the messageto the user based on a request from the user e) receiving a replymessage from the user via the virtual assistant; f) communicating thereply message to the server associated with the messaging application;and g) transmitting the reply message to the first device.

Provided is a computer implemented method to be carried out with aprocessor and a memory, comprising: a) receiving an initial message fromthe user to a “contact” on the first device via the virtual assistant;b) communicating the initial message to a server associated with themessaging application; c) transmitting the initial message to the firstdevice; and d) transmitting the initial message to a device associatedwith the contact.

Provided is a computer implemented method to be carried out with aprocessor and a memory, comprising: a) receiving an instruction from theuser to clear any unread messages from the first device; b)communicating the instruction to a server associated with the messagingapplication; c) transmitting the instruction to the first device; and d)deleting any unread messages on the device.

Provided is a smart phone (or other electronics devices mentionedabove), comprising: one or more processors; a memory, a touch screen;one or more programs, wherein the one or more programs are stored in thememory and configured to be executed by the one or more processors, theone or more programs including instructions to carry out the recitedsteps above.

Provided is a system for delivering one or more of a text message and anemail and a message from the messenger application to a user engaged inan activity, the system comprising a processor, a memory, and a touchscreen, the system configured to deliver the text message, the messagefrom the messenger application, and the email to the user by carryingout the recited steps above.

System

FIG. 1 is a use-case diagram showing a system 100, consistent with someembodiments, for messaging over a network by using a virtual assistant.

As illustrated in FIG. 1 , the frontend of the virtual assistant may bea virtual assistant physical virtual assistant device 105, such as ECHOmade available by Amazon.com Inc., of Seattle WA, or GOOGLE HOME madeavailable by Google LLC of Mountain View CA. The virtual assistantdevice 105 may have a microphone and a speaker for receiving verbalinput and generating audio output respectively. The virtual assistantdevice 105 may include a companion virtual assistant application(described below with respect to FIG. 3 ), which is an application thatconnects with a virtual assistant backend in the cloud (e.g., a virtualassistant server 140). A user 110 may take an affirmative step torequest, typically by voice, that data regarding his or her receivedmessages, typically from her smart phone 130 or other electronic device(e.g., desktop computer 120 or tablet device 125), be provided to theuser 110, e.g., read aloud.

In some embodiments, the virtual assistant device 105 may be in a“listening mode” wherein the companion virtual assistant application maybe activated based on receiving a particular verbal command, such as“hey google” or “hey Siri” or “Alexa.” After the virtual assistantapplication has been activated, the user 110 may request the virtualassistant application to 1) notify the user 110 regarding any messagesreceived by an electronic device of the user 110 (e.g., smart phone130), and/or 2) provide data associated with the messages (e.g., number,source, sender, body etc.) to user 110. For example, the data associatedwith the messages may be read aloud by virtual assistant device 105 inorder of receipt of the messages until all of the data has been providedto user 110.

In some embodiments, the emails, text messages, voice messages,voicemail, video messages, and/or messages from other applications arereceived by the smart phone 130 or other electronic device (e.g., 120,125, etc.) of user 110 via a communication network 115. For example, thecommunication network 115 may be the network of a mobile phone carrieror that of an internet carrier. Internet Message Access Protocol (IMAP)may be used as an Application Layer Internet protocol to communicatewith a remote mail server to access email. Text messages may becommunicated, for example, across the mobile carrier of the user 110using Short Message Service (SMS), which uses standardizedcommunications protocols to allow mobile smart phone 130 and/or otherelectronic devices (e.g., 120, 125, etc.) to exchange short textmessages. The SMS text message is “short” and may be limited to a numberof characters.

In some embodiments, the message type may be based on a source of themessage, for example, a messenger service such as Facebook messenger,WhatsApp messenger, Skype messenger, Twitter Direct Messenger, Tangomessenger, Viber Messenger, WeChat messenger, Instagram directmessenger, and iMessage (Apple chat). The messenger service is aninstant messaging service and software application which provides chatwith a friend on electronic devices that include a mobile carrierantenna, such as a mobile smart phone 130. The messenger service may bebuilt, for example, based on the open MQ Telemetry Transport (MQTT)network protocol that transports messages between devices.

In some embodiments, rather than using a dedicated virtual assistantdevice (e.g., virtual assistant device 105), the virtual assistant mayreside on a personal device of user 110, such as a smart watch of user110 or the smart phone 130 of user 110. However, the virtual assistantmay reside on desktop computer 120, tablet device 125, a vehicleoperating system and/or home devices/appliances (such as a television,refrigerator, etc.). The user 110 may request the virtual assistant onhis smart phone 130 to provide (e.g., read aloud) the message data formessages received at the smart phone 130 itself or at another electronicdevice (e.g., 125 or 130). The backend methods for processing therequest and the messages of user 110 may be implemented in the samemanner as described herein with respect to a dedicated virtual assistanthosting device, such as virtual assistant device 105.

In some embodiments, the message request of user 110 may be communicatedfrom the virtual assistant device 105 hosting the virtual assistantapplication to the cloud (e.g., to virtual assistant server 140 of cloudservices 135), which may be Amazon's web service (AWS) or another cloudbased service. The message request of user 110 may then be processed(e.g., voice to text and determination of intent of user 110) by thevirtual assistant server 140 before being communicated to a separatebackend server 145, which may also reside in the cloud (e.g., in cloudservices 135). The message request of user 110 may be communicated tothe backend server 145 because a “messages” intent was determined foruser 110 and backend server 145 hosts a “skill” which facilitatesmessaging via the virtual assistant (pingloud skill from messageLOUDInc.). Accordingly, backend server 145 may be referred to as the “skillbackend”. In the context of a virtual assistant, a “skill” is like anapp for Google Android or Apple IOS, for example, you may have skillsdeveloped to utilize the Amazon Alexa infrastructure. The Alexa SkillsKit is a collection of self-service APIs and tools that make it fast andeasy for developers to create new voice-driven capabilities (e.g.,messaging) for Alexa. For example, if user 110 wanted to book areservation at a restaurant, the user 110 could simply download areservation booking skill and then use it to book the reservation. Theskill may be hosted by the virtual assistant server 140 so thatcommunication with the backend server 145 is not required.

The communication of the message request from the virtual assistantserver 140 to the backend server 145 may also include the communicationof a token. A token is a private and/or encrypted key that allowslogging into a virtual assistance account of user 110 (e.g., anAmazon/Alexa account) from the backend server 145. The transmitted tokenmay be used to match the same user for two applications as the user 110logs into the virtual assistant application (e.g., Alexa companion app)hosted by virtual assistant device 105 and a messaging application(e.g., ping app from messageLOUD Inc.) running on a separate electronicdevice (e.g., smart phone 130) with the same Amazon user account. Thetoken may be used by backend server 145 to get user profile data (e.g.,from storage 155) for user 110 as per his/hers Amazon user account. Insome embodiments, an access token may be used to initially access Amazonprofile data for user 110 and a refresh token may subsequently be usedto refresh the token data after a specified time period, which limitsthe “lifetime” of the originally communicated access token, has expired.

In some embodiments, the token may be communicated from the backendserver 145 to a separate third party server 150 which hosts anapplication (e.g., Firebase from Google) configured to allow for a thirdparty to log into a virtual assistant user account of user 110 from thethird-party server 150 based on the token. The Firebase application mayprovide authentication to improve the sign-in and onboarding experiencefor end users, like user 110 in system 100. For example, the Firebaseapplication may provide an end-to-end identity solution for system 100by supporting interaction with email and password accounts, phoneauthentications, and diverse logins: Google, Twitter, Facebook, GitHub,etc.

In some embodiments, the above noted ping application (which is amessaging management application described, in part, in U.S. Pat. No.9,591,117 which is incorporated herein by reference in its entirety)running on smart phone 130, may check for newly received (e.g., sincethey were last checked) messages, and then if there are any, send dataassociated with the messages (e.g., source, sender, subject, body, etc.)to the skill backend server 145. In some embodiments, the backend servermay store the data associated with the messages for later access, forexample the message data may be stored in a storage 160 configured tostore the message data types. The backend server 145 may thencommunicate the message data to the virtual assistant server 140 whichmay host a program like Alexa Voice Service for voice processing (e.g.,text to audio). The virtual assistant server 140 may then communicatethe processed message data to the virtual assistant device 105 whichprovides (e.g., reads aloud) the messages.

In some embodiments the messages received at smart phone 130 may bereplied to after they have been communicated to user 110 by virtualassistant device 105. Alternatively, the user may generate an initialmessage (e.g., instead of a reply) to a “contact” of the user on smartphone 130. The virtual assistant device 105 may receive a spoken replyto at least one of the messages received by the smart phone 130, forexample, “Please tell Bob that I will meet him for dinner at 8 pm”.Therefore, the system 100 might match the reply to the message based ona sender of the message being “Bob” and/or a subject of the messagebeing “dinner”. Alternatively the system 100 might match the reply to acontact on smart phone 130 based on a name field of the contact being“Bob”. The virtual assistant device 105 may then transmit the spokenreply to the virtual assistant server 140 for voice processing, whereinthe voice processing includes generating text based on the reply (orinitial message) and then generating an audio version of the reply basedon the text. The virtual assistant device 105 may then receive and playthe generated audio so that user 110 may listen to the audio and approveof the generated audio version of the reply (or initial message) beforeit is sent. The virtual assistant device 105 may therefore wait for aspoken acknowledgment user 110 that indicates that the played audiomatches the spoken reply. The virtual assistant device 105 may thentransmit the text generated based on the reply (or initial message) tobackend server 145 based on receiving the acknowledgement.

In some embodiments, the backend server 145 may then transmit dataassociated with the generated reply (or initial message) text to thesmart phone 130. The messaging application (e.g., ping app.) maygenerate a reply (or initial) message based on the received dataassociated with the reply text and transmit the reply message to asender of the at least one message (or a contact), e.g., Bob.

User Interface

FIG. 2 is a block diagram illustrating an example of a messagingapplication 200, consistent with some embodiments, for messaging over acommunication network by using a virtual assistant.

In some embodiments, the messaging application 200 (e.g., ping app frommessageLOUD Inc.) may be running on smart phone 130, although as notedabove it may be running on another device (e.g., 120, 125, etc.). Adisplay 205 of smart phone 130 may show a user interface 210 (UI) forthe messaging application 200.

The UI 210 may present the user 110 with graphical menu choices such asa “contacts” choice 220 and a “settings” choice 230. The UI 210 may alsopresent the user 110 with instructions for setting up the messagingapplication 200 for use in conjunction with the virtual assistantapplication so that messages received at the smart pone 130 may beaccessed via the virtual assistant application. The instructions may bepresented based on user 110 having selected an option for instructionsto be shown when the messaging application 200 is launched.

In some embodiments, the UI 210 may present the following instructionsfor using the messaging application 200 together with the virtualassistant application: 1) log into the messaging application 200 with auser account of user 110 (e.g., an Amazon user account); 2) select themenu choice “Settings” 230 of the UI 210; 3) select a settings menuchoice for the “Virtual Assistant Application” (e.g., Alexa); and 4)select a menu choice(s) for linking the user account of user 110 (e.g.,Amazon account of user 110) to the virtual assistant application.

FIG. 3 is a block diagram illustrating an example of a virtual assistantapplication 300, consistent with some embodiments, for messaging over acommunication network by using a virtual assistant.

In some embodiments, a UI 310 of the virtual assistant application 300(e.g., the Alexa companion app) may be accessed via the smart phone 130of user 110 (or another device: 120, 125, etc.) over the communicationnetwork 115 or over a short range connection protocol (e.g., Bluetooth)if the smart phone 130 is physically close enough to the virtualassistant device 105 hosting the virtual assistant application. In someembodiments, the frontend virtual assistant device 105 may have its owndisplay (and/or other controls) through which the user 110 may interactwith the UI 310 of the virtual assistant application 300.

The display 205 of smart phone 130 may show the user interface 310 (UI)for the virtual assistant application 300. The UI 310 may present theuser 110 with graphical menu choices such as a “skills” choice 320 and a“settings” choice 330. The UI 310 may also present the user 110 withinstructions for setting up the virtual assistant application 300 foruse in conjunction with the messaging application 200 so that messagesreceived at the smart pone 130 may be accessed via the virtual assistantapplication 300. As noted above with respect to FIG. 2 , theinstructions may be presented based on user 110 having selected anoption for instructions to be shown when the virtual assistantapplication 300 is launched.

In some embodiments, the UI 310 may present the following instructionsfor using the virtual assistant application 300 together with themessaging application 200: 1) log into the virtual assistant application300 with the user account of user 110 (e.g., the same Amazon useraccount used to log into messaging application 200); 2) select the menuchoice “Settings” 330 of the UI 310; 3) select the menu choice “Skills”320 of the UI 310; 4) search for and then select a skills menu choicefor interacting with the messaging application 200 (e.g., the pingLoudskill); 5) select a skills menu choice for enabling the selected skill(e.g., the pingLoud skill); and, after the skill has been enabled, 6)select a settings menu choice to “Enable Notifications”.

Data Flow

FIG. 4 is a ladder diagram illustrating a method 400, as used in someembodiments, for accessing messages over a network by using a virtualassistant.

A system for accessing messages over a network by using a virtualassistant (e.g., system 100 of FIG. 1 ) may operate in different modes.For example: 1) Message Delivery upon Request: in this mode all newlyreceived texts/emails/messages are aggregated by messaging application200 (e.g., ping application) and placed in local storage on the smartphone 130 of user 110 (e.g., running the Android operating system byGoogle). The new messages are passed from smart phone 130 to the virtualassistant server 140 (e.g., backend Alexa AVS) when user 110 asksvirtual assistant device 105 (e.g., Echo device) for new messages (e.g.,from smart phone 130) to be read aloud. Once read, each message may bedeleted from the messaging application 200 local storage. Alternatively,the messages may be deleted from the messaging application 200 localstorage before they are even read/heard based on an instruction from theuser 110 to do so.

2) Notifications: in this mode (which is optional since the messages maybe requested whether or not a notification has been received) all newlyreceived texts/emails/messages are sent to the virtual assistant server140 in real time (e.g., as they are received by messaging application200 at smart phone 130) and stored there (e.g., securely encrypted instorage 160) for a limited period of time (e.g., a specified timeperiod). The virtual assistant device 105 (e.g., Echo device or otherpersonal/virtual assistant device) may flash a light to indicate that anotification has been received and is pending. The user 110 may see theflashing light on virtual assistant device 105 and ask it (as in mode 1above) to provide message data relating to the pending notification(e.g., number, sender, subject, body, etc.).

At operation 402, the user 110 may enable notifications regardingmessages received at an electronic device of user 110 (e.g., viamessaging application 200 on smart phone 130).

At operation 404, the messaging application 200 on the smart phone 130may send a notification to backend server 145 based on a message beingreceived.

At operation 406, the backend server 145 may store the message whichcaused the notification in its own storage (e.g., storage 160) based onthe messaging application 200 sending the message together with thenotification. The message placed in storage 160 by the backend server145 may be delivered to the user 110 after a request by the user 110 asdescribed below with respect to operations 420 and 426. In someembodiments, the message may reside in storage 160 in an encryptedformat. Enabled notifications may be a parameter that messagingapplication 200 uses to make a decision on whether to store the receivedmessage in local storage (e.g., in smart phone 130) or send the messageto backend server 145 in real time. In some embodiments, the storedmessage may only exist on the backend server 145 (e.g., in storage 160)until it is provided to user 110 or until a pre-determined period oftime has expired and it is deleted. As noted above, the messages may bedeleted from the messaging application 200 local storage before they areeven read/heard based on an instruction from the user 110 to do so, thisinstruction may also delete any copies of the message from the storage160.

At operation 408, the backend server 145 may send the notification tothe virtual assistant server 140 (e.g., Alexa AVS).

At operation 410, the virtual assistant server 140 may send thenotification to the virtual assistant device 105 (e.g., Echo device).

At operation 412, the virtual assistant device 105 may provide thenotification to the user 110 in any manner. For example, the virtualassistant device 105 may have a light that turns on and/or blinks inorder to indicate the presence of a pending event, such as a receivedmessage.

At operation 414, the user 110 may verbally request a virtual assistantfrontend (e.g., the virtual assistant application 300 on virtualassistant device 105) to access messages received at an electronicdevice of user 110 (e.g., received by messaging application 200 on smartphone 130).

At operation 416, the virtual assistant device 105 may transmit thespoken verbal request to a virtual assistant backend (e.g., virtualassistant server 140).

At operation 418, the virtual assistant server 140 may voice process theverbal request (e.g., verbal to text conversion and intentdetermination) and, upon determining a “messaging” intent, transmit theprocessed request to a skill backend server 145 that handles the backendprocessing for a messaging skill of the virtual assistant (e.g., tooperate in conjunction with the messaging application 200). A token maybe passed (e.g., with the processed request) to the backend server 145in order to verify user 110 to the messaging application 200 asdescribed below.

At operation 420, the backend server 145 may determine whether dataassociated with messages received by the smart phone 130 has alreadybeen received and stored (e.g., in storage 160 as noted above) and,based on a determination that it has been, proceed directly to operation432 (as described below). For example, the messaging application 200 mayhave made the decision on whether to store the received message in localstorage (e.g., in smart phone 130) or send the message to backend server145 in real time, for storage in storage 160, based on notificationshaving been enabled by user 110 as explained above.

At operation 422, the backend server 145 may transmit the messagerequest of user 110 to the messaging application 200 on the smart phone130. As noted above, the backend server 145 may use third party server150 (e.g., Firebase server) to communicate with the messagingapplication 200 on smart phone 130 regarding the messages that have beenrequested by user 110. Also as noted above, the Firebase application mayprovide authentication (e.g., checking token values) to improve thesign-in experience for user 110 in system 100.

At operation 424, the messaging application 200 on the smart phone 130may send message data associated with received messages to backendserver 145 based on receiving the message request from backend server145.

At operation 426, the backend server 145 may send the message dataassociated with the received messages to the virtual assistant server140 (e.g., Alexa AVS) for processing (e.g., text to audio conversion).

At operation 428, the virtual assistant server 140 may send theprocessed message data to the virtual assistant device 105 (e.g., Echodevice).

At operation 430, the virtual assistant device 105 may provide themessage data to the user 110 in any manner. For example, the virtualassistant device 105 may play audio based on the processed message datafor user 110.

At operation 432, the stored message(s) may be deleted from the backendserver 145 (e.g., in storage 160 or smart phone 130) after it isprovided to user 110 or after a pre-determined period of time hasexpired since storage (e.g., 24 hrs). As noted above, the messages mayalso be deleted before they are read/heard based on an instructionreceived from the user 110 by the virtual assistant device 105.

FIG. 5 is a ladder diagram illustrating a method 500, as used in someembodiments, for sending messages over a network by using a virtualassistant.

A system for replying to messages over a network by using a virtualassistant may resemble system 100 of FIG. 1 .

At operation 502, the user 110 may verbally generate an initial messageto a contact on smart phone 130 or a reply to at least one messagereceived at the electronic device of user 110 (e.g., received bymessaging application 200 on smart phone 130) via the virtual assistantfrontend (e.g., the virtual assistant application 300 on virtualassistant device 105).

At operation 504, the virtual assistant device 105 may transmit thespoken verbal initial message or reply to the at least one receivedmessage to a virtual assistant backend (e.g., virtual assistant server140).

At operation 506, the virtual assistant server 140 may voice process theverbal message/reply, wherein the processing may include verbal to textconversion wherein a “message or reply” intent determination is made andthen followed by text to audio conversion of the message or reply.

At operation 508, the virtual assistant server 140 may transmit thegenerated audio message or reply to the virtual assistant device 105.

At operation 510, the virtual assistant device 105 may play the audiomessage or reply.

At operation 512, the user 110 may verbally acknowledge that the audiomessage or reply matches the spoken message or reply.

At operation 514, the virtual assistant device 105 may transmit theverbal acknowledgement to the virtual assistant server 140 for voiceprocessing, e.g., verbal to text conversion and determination of amessage acknowledgement intent or a reply acknowledgement intent.

At operation 516, the virtual assistant server 140 may transmit theprocessed acknowledgement message or reply intent (e.g., text) to theskill backend server 145 that handles the backend processing for amessaging skill of the virtual assistant (e.g., to operate inconjunction with the messaging application 200). As noted above, a tokenmay be passed (e.g., with the voice processed request) to the backendserver 145 in order to verify user 110 to the messaging application 200.

At operation 518, the backend server 145 may generate data associatedwith the message or reply, e.g., reply type, reply recipient(s), replysubject, reply body, etc.

At operation 520, the backend server 145 may transmit the message orreply data to the messaging application 200 on smart phone 130. In someembodiments, the backend server 145 may store the message or reply datain a database (e.g., storage 160) for a period of time. As noted above,the backend server 145 may use a third party server (e.g., third partyserver 150) to interact with the messaging application 200.

At operation 522, the messaging application 200 on smart phone 130 maygenerate a message or reply based on the received message or reply data.For example, a reply type: e-mail, reply recipient: Bob@gmail.com, replysubject: “dinner” and reply body: “I will meet you at 8 pm”.

At operation 524, the messaging application 200 on smart phone 130 maytransmit the generated message or reply to an electronic deviceassociated with the recipient (e.g., Bob) of the message or reply (e.g.,recipient's smart phone 165).

Methods

FIG. 6 is a flow diagram illustrating a method 600, in accordance withembodiments, for accessing messages over a network by using a virtualassistant.

At operation 602, a virtual assistant frontend (e.g., the virtualassistant application 300 on virtual assistant device 106) may receive afirst request from the user 110 to check messages received at anelectronic device of user 110 (e.g., received by messaging application200 on smart phone 130).

At operation 604, the first request of user 110 may be transmitted tothe backend server 146 (e.g., because a “messages” intent was determinedfor the first request and backend server 146 hosts a “skill” whichfacilitates messaging via the virtual assistant frontend (pingloud skillfrom messageLOUD Inc.).

At operation 606, the backend server 146 may transmit the first requestof user 110 to the messaging application 200 on the smart phone 130. Asnoted above, the backend server 146 may use third party server 160(e.g., Firebase server) to communicate with the messaging application200 on smart phone 130 regarding the messages that have been requestedby user 110.

At operation 608, the backend server 146 may determine whether any dataassociated with messages received by the smart phone 130 has beenreturned from smart phone 130 in response to the first request. Themethod 600 may end based on a determination that no message data hasbeen returned by smart phone 130.

At operation 610, the backend server 146 may generate a response to thefirst request based on message data received from messaging application200 on the smart phone 130.

At operation 612, the backend server 146 may send the generated responseto the virtual assistant device 106 (e.g., Echo device).

At operation 614, the virtual assistant device 106 may provide themessage data to the user 110 in any manner. For example, the virtualassistant device 106 may play audio based on the generated response foruser 110.

FIG. 7A is a flow diagram illustrating example method 700A for voiceprocessing to access messages over a network by using a virtualassistant.

At operation 702A (which may follow from operation 602 of method 600 ofFIG. 6 ), the virtual assistant device 106 may determine whether thefirst request is a spoken verbal request. The method 700A may return tooperation 604 of method 600 based on a determination that the firstrequest is not a spoken request.

At operation 704A, the virtual assistant device 106 may transmit thespoken verbal request to a virtual assistant backend (e.g., virtualassistant server 140) based on a determination that the first request isa spoken verbal request.

At operation 706A, the virtual assistant server 140 may voice processthe spoken request (e.g., verbal to text conversion and intentdetermination) and then method 700A may proceed to operation 604 ofmethod 600.

FIG. 7B is a flow diagram illustrating example method 700B for voiceprocessing to access messages over a network by using a virtualassistant.

At operation 702B (which may follow from operation 610 of method 600 ofFIG. 6 ), the virtual assistant device 106 may determine whether theresponse is a text-based response. The method 700B may return tooperation 610 of method 600 based on a determination that the firstresponse is not a text-based response.

At operation 70413, the virtual assistant device 106 may transmit theresponse to a virtual assistant backend (e.g., virtual assistant server140) based on a determination that the response is a text-basedresponse.

At operation 706B, the virtual assistant server 140 may voice processthe text-based response (e.g., text to audio conversion) and then method700A may proceed to operation 612 of method 600.

FIG. 8 is a flow diagram illustrating a method 800, consistent with someembodiments, for sending messages by using a virtual assistant.

At operation 802 (which may follow from operation 614 of method 600 ofFIG. 6 ), the virtual assistant frontend (e.g., the virtual assistantapplication 300 on virtual assistant device 105) may receive an initialmessage from user 110 to a contact on smart phone 130 or a reply fromthe user 110 to at least one of the messages received at the electronicdevice of user 110 (e.g., received by messaging application 200 on smartphone 130).

At operation 804, the virtual assistant device 105 may transmit thespoken verbal initial message or reply to the at least one receivedmessage to a virtual assistant backend (e.g., virtual assistant server140).

At operation 806, the virtual assistant server 140 may voice process theverbal initial message or reply, wherein the processing may includeverbal to text conversion, “message or reply” intent determination andtext to audio conversion based on the text derived from the spokeninitial message or reply.

At operation 808, the virtual assistant device 105 may receive and thenplay aloud the audio initial message or reply for the user 110 toevaluate.

At operation 810, the user 110 may verbally acknowledge whether theaudio reply matches the spoken initial message or reply.

At operation 812, the virtual assistant device 105 may request (e.g.,via audio) that the user 110 repeat the initial message or reply basedon the user not confirming that the audio initial message or replymatches the spoken initial message or reply. For example, the user 110may state “No” or “not right” after hearing the audio initial message orreply:

At operation 814, the virtual assistant server 140 may transmit theprocessed initial message or reply (e.g., text derived from spokeninitial message or reply) to the skill backend server 145 that handlesthe backend processing for a messaging skill of the virtual assistant(e.g., to operate in conjunction with the messaging application 200). Asnoted above, a token may be passed (e.g., with the voice processedinitial message or reply) to the backend server 145 in order to verifyuser 110 to the messaging application 200.

At operation 816, the backend server 145 may generate data associatedwith the processed initial message or reply (e.g., reply type, replyrecipient(s), reply subject, reply body, etc.) and transmit the initialmessage or reply data to the messaging application 200 on smart phone130. In some embodiments, the backend server 145 may store the initialmessage or reply data in a database (e.g., storage 160) for a period oftime. As noted above, the backend server 145 may use a third partyserver (e.g., third party server 150) to interact with the messagingapplication 200 on smart phone 130.

At operation 818, the messaging application 200 on smart phone 130 maygenerate an initial message or reply message based on the receivedinitial message or reply data (e.g., a reply type: e-mail, replyrecipient: Bob@gmail.com, reply subject: “dinner” and reply body: “Iwill meet you at 8 pm”) and transmit the generated initial message orreply message to an electronic device associated with the recipient(e.g., Bob) of the reply or message (e.g., recipient's smart phone 165).

FIG. 9 is a flow diagram illustrating a method 900, consistent with someembodiments, for requesting message notifications by using a virtualassistant.

At operation 902, the user 10 may run the messaging application 200 onsmart phone 130, e.g., the ping application from messageLOUD Inc.

At operation 904, the user 110 may request to be notified at the virtualassistance device 105 of any messages received at smart phone 130. Theuser 110 may do this by enabling notifications via a menu option of a UIof the messaging application 200, as described above with respect toFIG. 2 .

At operation 906, the backend server 145 may determine whether anynotifications regarding messages received by the smart phone 130 hasbeen returned from smart phone 130 in response to the second request.The method 900 may end based on a determination that no notification hasbeen returned by smart phone 130 in response to the second request.

At operation 908, the backend server 145 may receive message data (e.g.,including any notifications and/or any messages) from smart phone 130.

At operation 910, the backend server 145 may store the received messagedata (e.g., in storage 160 as explained above) until they are requestedby user 110 or until a specified time period has expired (e.g., 48 hrs).

At operation 912, the backend server 145 may transmit thenotification(s) to the virtual assistant device 105 (e.g., Echo device).

At operation 914, the virtual assistant device 105 may provide thenotification to the user 110 in any manner. For example, the virtualassistant device 105 may flash a light to indicate that a notificationis pending for user 110.

Machine Architecture

FIG. 10 is a block diagram illustrating a machine in the example form ofa computer system 1000, within which a set or sequence of instructionsmay be executed to cause the machine to perform any one of themethodologies discussed herein, according to an example embodiment.

In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of either a serveror a client machine in server-client network environments, or it may actas a peer machine in peer-to-peer (or distributed) network environments.The machine may be an onboard vehicle system, wearable device, personalcomputer (PC), a tablet PC, a hybrid tablet, a personal digitalassistant (PDA), a mobile telephone, or any machine capable of executinginstructions (sequential or otherwise) that: specify actions to be takenby that machine. Further, while only a single machine is illustrated,the term “machine” shall also be taken to include any collection ofmachines that individually or jointly execute a set (or multiple sets)of instructions to perform any one or more of the methodologiesdiscussed herein. Similarly, the term “processor-based system” shall betaken to include any set of one or more machines that are controlled byor operated by a processor (e.g., a computer) to individually or jointlyexecute instructions to perform methods as discussed herein.

Example computer system 1000 includes at least one processor 1002 (e.g.,a central processing unit (CPU), a graphics processing unit (GPU) orboth, processor cores, compute nodes, etc.), a main memory 1004 and astatic memory 1006, which communicate with each other via a link 1008(e.g., bus). The computer system 1000 may further include a videodisplay unit 1010, an alphanumeric input device 1012 (e.g., a keyboard),and a user interface (UI) navigation device 1014 (e.g., a mouse). In oneembodiment, the video display unit 1010, input device 1012 and UInavigation device 1014 are incorporated into a touch screen display. Thecomputer system 1000 may additionally include a storage device 1016(e.g., a drive unit), a signal generation device 1018 (e.g., a speaker),a network interface device 1020, and one or more sensors 1022, such as aglobal positioning system (GPS) sensor, accelerometer, gyrometer orother sensor.

The storage device 1016 includes a machine-readable medium 1024 on whichis stored one or more sets of data structures and instructions 1026(e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1026 mayalso reside, completely or at least partially, within the main memory1004, static memory 1006, and/or within the processor 1002 duringexecution thereof by the computer system 1000, with main memory 1004,static memory 1006, and processor 1002 comprising media.

While the machine-readable medium 1024 is illustrated in an exampleembodiment to be a single medium, the term “machine-readable medium” mayinclude a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more instructions 1026. The term “machine-readable medium”shall also be taken to include any tangible medium that is capable ofstoring, encoding or carrying instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present disclosure or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude volatile or non-volatile memory, including but not limited to,by way of example, semiconductor memory devices (e.g., electricallyprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM)) and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions 1026 may further be transmitted or received over acommunications network 1028 using a transmission medium via the networkinterface device 1020 utilizing any one of a number of well-knowntransfer protocols (e.g., HTTP). Examples of communication networksinclude a local area network (LAN), a wide area network (WAN), theInternet, mobile telephone networks, plain old telephone (POTS)networks, and wireless data networks (e.g., Wi-Fi, 3G, and 4G LTE/LTE-Aor WiMAX networks). The term “transmission medium” shall be taken toinclude any intangible medium that is capable of storing, encoding, orcarrying instructions for execution by the machine, and includes digitalor analog signals or other intangible medium to facilitate communicationof such software.

Example computer system 1000 may also include an input/output controller1030 to receive input and output requests from the at least one centralprocessor 1002, and then send device-specific control signals to thedevice they control. The input/output controller 1030 may free the atleast one central processor 1002 from having to deal with controllingeach separate kind of device.

APPENDIX Business Rules/in-Use Commands:

For every command below, add a version with the word ‘please’ before it.Once Alexa is reading messages the user can say ‘stop’ to stop readingmessages.

-   -   Alexa, stop    -   Alexa, stop reading    -   Alexa, stop reading messages    -   When Alexa is reading aloud, the user can say:    -   Alexa, archive this message    -   Alexa, archive    -   Alexa, delete this message    -   Alexa, delete    -   Alexa, skip this message    -   Alexa, skip    -   Alexa, mark as read    -   Alexa, mark as unread    -   Alexa, next message    -   Alexa, next

Alexa Commands:

-   -   Do I have any new emails    -   Do I have any new email    -   Do I have any new messages    -   Do I have any new text messages    -   Do I have any new sms's    -   Do I have any new WhatsApp messages    -   Do I have any new WhatsApps    -   Do I have any new Facebook messages    -   Do I have any emails    -   Do I have any email    -   Do I have any messages    -   Do I have any text messages    -   Do I have any sms's    -   Do I have any WhatsApp messages    -   Do I have any WhatsApps    -   Do I have any Facebook messages        *The answer to these is: “You have X unread <message type>.”    -   Read my emails    -   Read my email    -   Read my messages    -   Read my texts    -   Read my WhatsApp messages    -   Read my WhatsApps    -   Read my sms's    -   Read my Facebook messages        *The answer to this is to simply start reading any unread        <message types>.    -   Read emails    -   Read email    -   Read messages    -   Read texts    -   Read WhatsApp messages    -   Read WhatsApps    -   Read sms's    -   Read Facebook messages        *The answer to this is to simply start reading any unread        <message types>.    -   How many new texts do I have?    -   How many new sms's do I have?    -   How many new emails do I have?    -   How many new Facebook messages do I have?    -   How many new WhatsApp messages do I have?    -   How many new WhatsApps do I have?        *The answer to these is: “You have X unread <message type>.”

What is claimed is: 1-20. (canceled)
 21. A computer implemented method,comprising: receiving, by a virtual assistant program hosted by avirtual assistant device, a first request to check for messages receivedby an electronic device; transmitting the first request to a backendserver for processing; requesting from a messaging application on theelectronic device, by the backend server, stored data associated withthe messages; receiving the data associated with the messages andgenerating a response to the first request based on the received dataassociated with the messages; transmitting the response to the virtualassistant program; and communicating the response by the virtualassistant device.
 22. The method of claim 21, wherein the dataassociated with the messages includes at least one of a number, asource, a date, a time or a body of the messages that have been receivedby the electronic device.
 23. The method of claim 1, wherein the firstrequest is spoken to the virtual assistant device and communicating theresponse includes playing audio by the virtual assistant device.
 24. Themethod of claim 21, wherein the virtual assistant device comprises atleast one of a smart phone, a smart watch, a personal computer, a tabletcomputer, a vehicle or a home appliance and the messages comprise atleast one of emails, text messages, voice messages, voicemail or videomessages.
 25. The method of claim 21, further comprising: transmittingthe first request to a server associated with the virtual assistantprogram for voice processing, wherein the voice processing includesgenerating text; and transmitting the response to the server associatedwith the virtual assistant program for text processing, wherein the textprocessing includes generating audio.
 26. The method of claim 21,further comprising: receiving, by the virtual assistant device, a spokenreply to at least one of the messages received by the electronic device;transmitting the reply to a server associated with the virtual assistantprogram for voice processing, wherein the voice processing includesgenerating text based on the reply and generating audio based on thetext; receiving and playing the generated audio by the virtual assistantdevice; receiving, by the virtual assistant device, a spokenacknowledgment that the played audio matches the spoken reply; andtransmitting the text to the backend server based on receiving theacknowledgement.
 27. The method of claim 26, further comprising:transmitting to the electronic device, by the backend server, dataassociated with the text based on the reply; generating a reply message,by the messaging application on the electronic device, based on thereceived data associated with the reply; and transmitting the replymessage to a sender of the at least one message.
 28. The method of claim27, wherein the spoken reply to at least one of the messages received bythe electronic device comprises an initial message to a contact on theelectronic device and transmitting the reply message to a sender of theat least one message comprises transmitting the initial message to thecontact on the electronic device.
 29. The method of claim 21, furthercomprising: transmitting a token to the backend server with the firstrequest; and using the token to match a user that is logged into thevirtual assistant program and the messaging application-to a same useraccount.
 30. The method of claim 21, further comprising: receiving, bythe virtual assistant program, a second request to be notified of theelectronic device receiving at least one of the messages; instructing,by the backend server, the messaging application on the electronicdevice to automatically transmit data associated with the at least onemessage based on receiving the second request from the virtual assistantprogram; receiving the at least one message by the electronic device andtransmitting data associated with the at least one message to thebackend server; storing the data associated with the at least onemessage; transmitting a notification to the virtual assistant programbased on receiving the data associated with the at least one message;and communicating the notification by the virtual assistant device. 31.The method of claim 30, further comprising: generating the response tothe first request based on the stored data; and at least one of deletingthe stored data based on the communication of the response or deletingthe stored data based on a specified time period.
 32. The method ofclaim 21, further comprising: receiving a deletion request for unreadmessages on the electronic device; transmitting the deletion request tothe backend server for processing; requesting, by the backend server,that the messaging application on the electronic device delete dataassociated with the unread messages; and deleting data associated withthe unread messages by the electronic device.
 33. A system formessaging, comprising: a virtual assistant device to: receive a firstrequest to check for multiple types of messages received by anelectronic device; and transmit the first request to a backend serverfor processing; the backend server to: request stored data associatedwith the multiple types of messages from a messaging application on theelectronic device; receive the data associated with the messages;generate a response to the first request based on the received data; andtransmit the response to the virtual assistant device; and the virtualassistant device to communicate the response.
 34. The system of claim33, further comprising: the virtual assistant device to: receive aspoken reply to at least one of the messages received by the electronicdevice; transmit the reply to a server associated with the virtualassistant device for voice processing, wherein the voice processingincludes generating text based on the reply and generating audio basedon the text; receive and play the generated audio; receive a spokenacknowledgment that the played audio matches the spoken reply; andtransmit the text to the backend server based on receiving theacknowledgement.
 35. The system of claim 34, further comprising: thebackend server to: transmit data associated with the text to theelectronic device; and the messaging application on the electronicdevice to: generate a reply message based on the data associated withthe text; and transmit the reply message to a sender of the at least onemessage.
 36. The system of claim 33, further comprising: the virtualassistant device to receive a second request to be notified of theelectronic device receiving at least one of the messages; the backendserver to instruct the messaging application on the electronic device toautomatically transmit data associated with the at least one receivedmessage based on receiving the second request from the virtual assistantdevice; the electronic device to receive the at least one message andautomatically transmit data associated with the at least one message tothe backend server, the backend server to store the data associated withthe at least one message and transmit a notification to the virtualassistant device based on receiving the data associated with the atleast one message; and the virtual assistant device to communicate thenotification.
 37. A non-transitory computer readable medium (CRM)storing instructions which, when executed, cause one or more processorsto perform operations, comprising: receiving, by a virtual assistantprogram hosted by a virtual assistant device, a first request to checkfor messages received by an electronic device; transmitting the firstrequest to a backend server for processing; requesting from a messagingapplication on the electronic device, by the backend server, stored dataassociated with the messages; receiving the requested data andgenerating a response to the first request; transmitting the response tothe virtual assistant program; and communicating the response by thevirtual assistant device.
 38. The CRM of claim 37, further comprisinginstructions which, when executed, cause the processors to performoperations, comprising: receiving, by the virtual assistant device, aspoken reply to at least one of the messages received by the electronicdevice; transmitting the reply to a server associated with the virtualassistant program for voice processing, wherein the voice processingincludes generating text based on the reply and generating audio basedon the text; receiving and playing the generated audio by the virtualassistant device; receiving, by the virtual assistant device, a spokenacknowledgment that the played audio matches the spoken reply; andtransmitting the text to the backend server based on receiving theacknowledgement.
 39. The CRM of claim 38, further comprisinginstructions which, when executed, cause the processors to performoperations, comprising: transmitting to the electronic device, by thebackend server, data associated with the text based on the reply;generating a reply message, by the messaging application on theelectronic device, based on the received data associated with the reply;and transmitting the reply message to a sender of the at least onemessage.
 40. The CRM of claim 37, further comprising instructions which,when executed, cause the processors to perform operations, comprising:receiving, by the virtual assistant program, a second request to benotified of the electronic device receiving at least one of themessages; instructing, by the backend server, the messaging applicationon the electronic device to automatically transmit data associated withthe at least one received message based on receiving the second requestfrom the virtual assistant program; receiving the at least one messageby the electronic device and automatically transmitting data associatedwith the at least one message to the server, storing the data associatedwith the at least one message; transmitting a notification to thevirtual assistant program based on receiving the data associated withthe first message; and communicating the notification by the virtualassistant device.